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COMPACT MICROSCOPE IMAGING SYSTEM WITH INTELLIGENT 

CONTROLS 



ORIGIN OF THE INVENTION 

The invention described herein was made by an employee of the United States 
Government and may be used by or for the Government for governmental purposes 
10 without payment of any royalties thereon or therefor. 




FIELD OF THE INVENTION 



The invention relates to a Compact Microscopic Imaging System (CMS) with 
15 intelligent controls. Specifically, the invention relates to a CMIS that provides 

techniques for scanning, identifying, detecting and tracking microscopic changes and 
selected characteristics or features of various surfaces included, but not limited to, cells, 
spheres, and manufactured products subject to difficult-to-see imperfections. The CMIS 
provides intelligence to allow experiments to be conducted without the need of constant 
20 or even human monitoring. 



BACKGROUND OF THE INVENTION 



Microscope experiments are typically conducted by humans utilizing microscopes 
25 that provide enlarged images of minute objects or specimens being analyzed. The 

successful results of these experiments are dependent upon the concentration, endurance, 
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and constant monitoring of a human operating the microscope. Sometimes the human 
operator of the microscope on a daily basis loses concentration, becomes fatigued as the 
day goes on, or becomes distracted with all these factors contributing to unsuccessful 
experiments. It is desired that a system be provided that allows experiments to be 
conducted using microscope, but without the need of intervention by a human. 

OBJECTS OF THE INVENTION 

It is a primary object of the present invention to provide a system that allows 
experiments to be conducted utilizing microscopes, but without the need of human 
intervention. 

It is another object of the present invention to allow experiments to be conducted 
without the need of human intervention and without the need of constant monitoring. 

It is a further object of the present invention to provide a system for analyzing 
microscope experiments automatically and remotely. 

It is a further object of the present invention to provide a system that utilizes 
intelligence and controls the operation of the system by the use of adaptive neural 
networks. 



Still further, it is an object of the present invention to provide for a system that 
scans to find areas of interest, focuses and acquires images automatically. 

In addition, it is an object of the present invention to provide a system to evaluate 
5 surfaces of devices, such as semiconductors, to ascertain the quality thereof 

SUMMARY OF THE INVENTION 

The present invention is directed to a Microscope Imaging System (CMIS) that 
10 utilizes intelligent controls and provides techniques for scanning, identifying, detecting 
and tracking microscopic changes in selected characteristics or features of various objects 
including, but not limited to, cells, spheres and manufactured products subject to 
difficult-to-see imperfections. 

15 In one embodiment, an imaging system for examining specimens is provided 

comprising: a video microscope comprising a holder for specimens and a camera located 
so as to be focussed on the holder. The imaging system further comprises a light source 
operatively connected to the video microscope, and a robotic system operatively 
connected to and for positioning the video microscope. The imaging system further 

20 comprises a computer operatively connected to the camera and to the robotic system. 
The computer has operating programs comprising routines for providing machine vision 
techniques for scanning, identifying, detecting and tracking selected characteristics and 
features of the specimen being analyzed. 
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BRIEF DESCRIPTION OF THE DRAWINGS 



The features and advantage of the invention, as well as the invention itself, will be 
better understood with reference to the following description when considered in 
5 conjunction with the accompanying drawings, wherein like reference numbers designate 
identical corresponding parts thereof and wherein: 

Fig. 1 is a block diagram of the imaging system of the present invention; 

Fig. 2 is a flow chart of one embodiment of the present invention; 

Fig. 3 illustrates a colloid image related to the embodiment of Fig. 2 and shows 
10 horizontal growth; 

Fig. 4 illustrates a colloid image related to the embodiment of Fig. 2 and shows 
vertical growth; 

Fig. 5 illustrates a typical colloid image; 

Fig. 6 is a typical frame averaged image; 
15 Fig. 7 is composed of Figs. 7A and 7B, which cumulatively illustrate a flow chart 

for another method of the present invention; 

Fig. 8 illustrates the result of brightness slicing on a frame-average image related 
to the embodiment of Fig. 7; 

Fig. 9 illustrates an image of a dilation treated image combined with optimal 
20 threshold related to the embodiment of Fig. 7; 

Fig. 10 is an image treated with a find particle algorithm related to the 
embodiment of Fig. 7; 
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Fig. 1 1 is an image illustrating the interface analysis associated with the 
embodiment of Fig. 7; 

Fig. 12 illustrates a typical colloid image showing related to vertical growth 
associated with the embodiment of Fig. 7; 
5 Fig. 13 illustrates the image showing the interface analysis related to vertical 

growth associated with the embodiment of Fig. 7; 

Fig. 14 is composed of Figs. 14 A and 14B that cumulatively illustrate a flow chart 
related to a further method of the present invention; 

Fig. 15 illustrates an out-of-focus image related to the method of Fig. 14; 
10 Fig. 16 illustrates Fast Fourier Transform (FFT) of an out-of-focus image related 

to the method of Fig. 14; 

Fig. 17 illustrates an in-focus image related to the method of Fig. 14; 

Fig. 18 illustrates a FFT of an in-focus image related to the method of Fig. 14; 

Fig. 19 is composed of Figs. 19A and 19B that cumulatively illustrate a flow chart 
15 related to a still further method of the present invention; 

Fig. 20 illustrates an image of a candidate specimen related to the method of Fig. 

19; 

Fig. 21 illustrates a histogram that as received Sobel filtering operation all related 
to the method of Fig. 19; 
20 Fig. 22 illustrates an image after receiving variance filtering related to the method 

of Fig. 19; 

Fig. 23 illustrates an image after receiving Sobel and variance filtering techniques 
all related to the method of Fig. 19; 



5 



Fig. 24 illustrates an image after having all objects therein identified and related 
to the method of Fig. 19; 

Fig. 25 illustrates an image of a second candidate specimen related to the method 
of Fig. 19; 

5 Fig. 26 illustrates the result of the image of Fig. 25 after having received the 

operation of the method of Fig. 19; 

Fig. 27 illustrates a flow chart related to another method of the present invention; 
Fig. 28 illustrates the original cell image related to the method of Fig. 27; 
Fig. 29 illustrates an image processed by the method of Fig. 27; 
10 Fig. 30 illustrates a image that had been labeled process by the method of Fig. 27; 

Fig. 3 1 illustrates an original image with the cells labeled and associated with the 
method of Fig. 27; 

Fig. 32 is a flow chart related to an additional method of the present invention; 
Fig. 33 is an image of an ideal case surface related to the method of Fig. 32; 
I 5 Fig. 34 illustrates a defective surface detected by the method of Fig. 32; 

Fig. 35 illustrates another defective surface detected by the method of Fig. 32; 
Fig. 36 illustrates an ideal surface of another semiconductor device related to the 
method of Fig. 32; 

Fig. 37 illustrates a second defective surface detected by the method of Fig. 32; 

20 and 

Fig. 38 illustrates another defective surface detected by the method of Fig. 32. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 



Referring to Fig. 1, there is shown a block diagram of Compact Microscope 
Imaging System 10 of the present invention particularly suited to perform microscope 
5 analysis. 



As will be further described, the Compact Microscope Imaging System (CMIS) 
10, sometimes referred to herein simply as an imaging system, is a diagnostic tool with 
intelligent controls for use in space, industrial, medical, and security applications. The 
10 CMIS 10 can be used in-situ with a minimum amount of user intervention and it can be 
used to scan, find areas of interest, focus, and acquire images all automatically. 



The CMIS 10 provides machine vision techniques in order to achieve the goal of 
being an intelligent microscope capable of several automated tasks and able to reproduce 
15 human microscope system interactions with accuracy and repeatability. The CMIS 10 
may be used to replace what humans do with microscopes on a daily basis. Unlike 
human operation, the CMIS 10 will not lose concentration, become fatigued as the day 
goes on, or become distracted like a human might. 

20 A wide variety of applications that can take advantage of the CMIS 10 having 

intelligent controls for usage in fields such as materials science, fluids science, life 
sciences, biotechnology and production monitoring for quality control where human 
visualization of a cell or surface is difficult. These include experiments involve dendrite 
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growth, transparent system coarsening and sintering, soil mechanics (including lunar and 
Martian experiments), colloid science, magneto and electro-rheological fluids, details of 
flow around bubbles, particle pushing and engulfment, foam behavior, protein and zeolite 
crystal growth, examination of living systems and microscopic or extremely small defects 
in high value parts. Large numbers of multiple cell experiments require microscopy for 
in-situ observations; this is feasible with the CMIS 10. 

In general, the CMIS 10 provides machine vision techniques characterized as 
follows: (1) adaptive thresholdings which optimally determines the best method to 
achieve the image contrast needed for image processing; (2) auto-focus which provides 
the ability to focus on a sample automatically utilizing a Cartesian robot system; (3) auto- 
imaging scanning which enables a microscope to scan left, right, up, down, back, and 
forth within a sample in order to find the object of interest; (4) object identification and 
classification which enables the finding, classifying and labeling of objects in a pre- 
determined area of interest; (5) motion detection which enables the observing and 
quantifying of the movement of objects in a pre-determined area of interest; and (6) 
transition mapping which enables the detection of small transitions between groups of 
cells used for microscope experiments such as order/disorder, large objects/small objects, 
light/dark regions and movement/non-movement. Transition mapping is particularly 
helpful in determining the exact transition location of the quasi-zone, which is in-between 
two phases in a microscope image. 
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To further appreciate the benefits of the present invention the CMIS 10 will be 
further described hereinafter for four (4) methods which are: (1) interface detection in 
colloidal hard spheres systems; (2) auto-focus for compact microscope systems; (3) cell 
identification and classification for biomedical applications and (4) surface defect 
5 classification. The overall operation may be further described with reference to Fig. 1 . 

The CMIS 10 comprises a video microscope 12, which, in turn, comprises a 
holder 14 and a camera 16 located, so as to be focused on the holder 14. The CMIS 10 
further comprises light source 18 that is operatively connected to the video camera 12 via 
10 signal path 20 and provides for illumination of any specimens being held in the holder 14. 
The CMIS 10 further comprises a robotic system 22 response to electrical signals and 
which is operatively connected to the video microscope 12 by way of control path 24 and 
preferably by way of platform 26 upon which the video microscope 12 resides. The 
robotic system 22 preferably further comprises a micro-positioner 22A. 

15 

The CMIS 10 further comprises a computer 28 that is operatively connected to the 
camera 16 by way of an image acquisition board 30 and signal path 32. The computer 28 
further provides electrical signals and is operatively connected to the robotic system 22 
by way of a robotic control board 34 and signal path 36. 

20 

The operation of the computer 28 is controlled by its operating programs 36, 
which are comprised of machine vision routines 38, algorithms 40, and neural network 
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routines 42. The operating programs 36 provide visual verification of the results of the 
performed testing, to be described, on display terminal 44, via signal path 46. 

The video microscope 12 is preferably compact and may be of the type that are 
made commercially available by various companies such as Fuji, Sony, Edmund 
Scientific and Zeiss. The camera 16 is preferably comprised of coupled devices and the 
camera 16 feeds the data to the acquisition board 30 by way of the signal path 32. The 
camera 16 provides an output representative of the electronic image of the specimen 
being held in holder 14. 

The light source 18 is preferably a fiber-optic, which may be of the type that are 
made commercially available by various companies, such as Cuda Products, Thor Labs 
and Semiconductor International. The robotic system 22 is preferably of a Cartesian type 
and provides three (3) dimensional positioning of the video microscope 12 which, in turn, 
provides three (3) dimensional positioning of the specimen being held in the holder 14, as 
well as the camera 16. The robotic system 22 may be of the type that are made 
commercially available by various companies, such as Parker Hanifin, Dadel Inc., and 
Physic Instruments. The operating programs 36 residing in the computer 28 further 
comprise routines (known in the art) for controlling the robotic system which, in turn, 
control the positioning of the video microscope 12. 

The algorithms 40, to be further described hereinafter, operatively cooperate with 
routines that are part of a machine vision routines that provide for scanning, identifying, 
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detecting and tracking selected characteristics and features of the many specimens 
capable of being analyzed by the CMIS 10. 

The neural network routines are of a conventional type that consists of a set of 
5 elements that start out connected in a random pattern, and based upon operational 
feedback, are molded into patterns required to generate the required results to be 
described hereinafter. These neural network routines operatively control the robotics 
system 22, as well as diagnose and provide image processing and pattern recognition 
related to the machine vision routines 38. 

10 

Adaptive neural networks are essentially probabilistic computations or techniques 
that are used to adapt neural networks for automated control tasks. A basic neural 
network approach is to accept several inputs (N) and based on an error relationship 
produce a finite set of outputs (M). The difficulty in using basis neural networks is that 

15 the results can be unpredictable and the neural network may fail to yield an optimal 
solution if conditions suddenly change. However, when adaptive neural networks are 
implemented as for the present invention, the "best case" scenario is optimized by a set of 
learning behaviors built into the neural network. Just as humans adapt to a changing 
environment, so can an adaptive neural network. Such is the case in the practice of the 

20 present invention with regard to the technique of auto-focus, auto-image scanning and the 
detection of defects. Rather than have several inputs mapped to a finite set of outputs 
(M), an adaptive neural network takes several inputs (N) and maps them to a set of 
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outputs (M+l), with the (+1) referring to a new case that can be optimized/defined in the 
future. 

The practice of the present invention uses inputs or features that a human would 

5 use to identify, detect and classify objects, cells and areas of interest. These are 

represented by the initial sets of images that are received. The practice of the invention 
uses image processing techniques and creates a relational database of the features 
identified by the humans. These features serve as a baseline input feature set and are 
used to train/teach the adaptive neural network preferably used in the practice of the 

10 invention. The practice of the invention develops a machine vision technique that is used 
to identify the input feature set, but also has the ability to learn how to adapt to various 
changing inputs to produce an optimal output. For example, let's take a group of cells 
that have 4 defects in each cell. A standard neural network technique, as well as an 
adaptive neural network technique, will identify only cells with 4 defects. Now for 

15 example, take a group of cells with 7 defects in them. A standard neural network 

technique will output no cells identified or label all cells as having only 4 defects, which 
would be a gross misrepresentation of the data. However, an adaptive neural network 
technique practiced by the present invention outputs cells with 4, 5, 6, or 7 defects since 
it has the ability to reason or adapt to the changing number of defects. One of the 

20 machine vision routines is related to interface detection of colloidal hard spheres systems. 

Microscopic hard spheres in a colloid suspension have been used for studying 
atomic phase changes from disorder to order or from liquid to solid. It is my 
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understanding that current microscope image analysis techniques are not able to 
distinguish between these densely packed phases within microscope images, which have 
similar grayscale value properties and are mainly characterized by degrees of randomness 
or order. The current technique for identifying the phased change boundaries involves 
manually identifying the transitions, which is very time consuming and inefficient. The 
present invention provides a method that overcomes these difficulties. More particularly, 
the present invention provides a technique that accurately detects the interface of 
colloidal hard spheres experiments automatically and without human intervention and 
may be further described with reference to Fig. 2. 

Fig. 2 illustrates an overall flow chart for a routine 48 for interface detection of 
colloidal hard spheres. The routine 48 is comprised of steps represented by segments 50, 
52, 54, 56, 58, and 60 shown in Fig. 2, some of which serve as program segments. 

In general, a colloid system, examined by the practice of the present invention, 
consists of two phases, a solid or ordered phase and a liquid or disordered phase. There 
is a quasi-zone in between these two phases, which has to be analyzed in order to 
determine an accurate interface. Depending on the orientation of the sample, the practice 
of the present invention provides an interface technique to analyze the following two 
cases: (1) horizontal crystal growth from left to right as shown in Fig. 3, and (2) vertical 
crystal growth from bottom to top as shown in Fig. 4. 

If the solid/liquid interface is oriented horizontally (the crystal is growing across 
the screen), the rightmost portion of this object contains the contour of the solid/liquid 
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interface generally identified by reference number 62 of Fig. 3. If the solid/liquid 
interface is oriented vertically (the crystal is growing up the screen), the top portion of 
this object contains the contour of the solid/liquid interface generally identified by 
reference number 64 of Fig. 3. 

5 

Based on kinetic theory, it is that matter is made of molecules in continual 
random motion. In a crystalline or solid state, these molecules are held in place by 
attractive forces and tend to vibrate around a fixed position. In a liquid state, the 
molecules have a greater range of motion. With this concept in mind, it is expected that 
10 the molecules in the liquid portion of the system exhibit a higher degree of motion than 
those in the solid. This higher degree expectation allows the present invention to use 
frame averaging to visually separate solid from liquid regions. 



The method of Fig. 2 averages a series of frames performed by one of algorithms 
15 40 by summing each individual pixel over each frame in the series and dividing by the 
number of frames. In a colloidal suspension, the spheres in the "liquid" region have more 
freedom of movement than the spheres in the solid region. When multiple frames are 
averaged, the liquid region tends to become blurry or grayed out while the solid region 
maintains a higher level of contrast and may be represented by the below expression: 

n 

i,j: row and column position of the pixel in a frame (image) 
n: number of frames 
I: input pixel, 
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O: output pixel 



Figs. 5 and 6 give examples of a typical image and the averaged image over a 
series of frames, wherein the liquid and solid portions of Fig. 5 are respectively generally 
identified by reference numbers 66 and 68 and wherein the liquid and solid portions of 
Fig. 6 are respectively generally identified by reference numbers 70 and 72. Note the 
loss of contrast in the liquid portion of the averaged image of Fig. 6. The overall 
operation of the method of interface detection of colloid hard surfaces may be further 
described with again reference to Fig. 2. 

As seen in Fig. 2, the method for interface detection of colloidal hard sphere is 
initiated by segment 50, which signifies the placing of the colloidal hard sphere 
specimen, having a solid/liquid interface to be examined, in the holder 14. Upon such 
placement, the method of Fig. 2 passes control to program segment 52, by way of signal 
path 74. 

The program segment 52 causes the video microscope 12 to generate an electronic 
image, serving as a frame of the colloidal specimen. Upon such generation by program 
segment 52, control is passed to program segment 54, by way of signal path 76. 

The program segment 54 examines the frame of the electronic image to determine 
if the solid/liquid interface has a horizontal or vertical growth previously discussed. 
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Upon completion, program segment 54 passes control to program segment 56, by way of 
signal path 78. 

The program segment 56 generates a series of frames of electronic images of the 
examined specimen and then passes control to program segment 58, by way of signal 
path 80. 

The program segment 58 performs averaging of the series of frames to provide an 
imaged averaged over the series of frames so as to separate the solid/liquid portions of 
the electronic image. Upon completion, program segment 58 passes control to program 
segment 60, via signal path 82. 

The program segment 60 examines, by displaying on terminal 44, the averaged 
image to identify the liquid portions by blurred or grayed out regions of the averaged 
image and to identify the solid portions as clear regions with contrast therein. 

Another embodiment for providing a routine for interface detection of colloidal 
hard surfaces, may be further described with reference to Fig. 7, which is composed of 
Figs. 7A and 7B, wherein Fig. 7A illustrates the overall method 84 comprised of 
elements 86, 88, 90, 92, 94, 96, and 98, and wherein Fig. 7B illustrates the overall 
method 84 as further comprising elements 100, 102, 104, 106, and 108. The program 84 
of Fig. 7 is different than the program 48 of Fig. 2 in that it provides for a brightness 
slicing technique. 
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The brightness slicing technique is used to create a very high-contrast image that 
specifically highlights a particular object of interest and works well in separating objects 
that have either light or dark values from a background with intermediate values. In the 
5 averaged image shown in Fig. 6, previously discussed, the ordered or solid portion of the 
image is made up of light and dark values, while the disordered or liquid portion of the 
image has intermediate gray values. As will be further described, the brightness slicing 
technique is performed on a series of frame-averaged images and when applied, the result 
will be a solid region highlighted in white against a black background, thereby making it 
10 quite easily to determine the solid-liquid interface. Fig. 8, to be further described 

illustrates a typical colloid image utilizing the brightness slicing technique. It should be 
noticed how there is now a distinct separation pattern from the solid and liquid regions of 
the image respectively generally identified by reference numbers 110 and 112. 

15 Brightness slicing employed by the method of Fig. 7 is a double binary contrast 

enhancement operation, serving as another algorithm 40, where pixel brightness' s below 
a lower threshold and above an upper threshold are set to white, while brightness' s 
between the two thresholds are set to black. This results in the liquid or blurry region 
being set to black while the solid region is set to white and may be represented by the 

20 below expression: 

=1, otherwise 
ai: lower threshold value. 



a 2 : upper threshold value. 

The values of the quantities ai and a 2 are dependent upon the experiment being 
conducted and more importantly, a light or a dark background. The range of the 
quantities, ai and a 2 , can be between O (black) and 255 (white). The brightness slicing 
technique may be further described with again reference to Fig. 7. 

As seen in Fig. 7 A, the overall routine 84 for interface detection of colloidal hard 
surfaces employing the brightness slicing technique is initiated by segment 86 which 
signifies the placing of colloidal hard sphere specimen in holder 14. Upon such 
completion, the method 84 passes control to program segment 88, by way of signal path 
114. 

Program segment 88 causes the video microscope 12 to generate an electronic 
image, serving as a frame, of the colloid specimen and then passes control to program 
segment 90, via signal path 1 16. 

Program segment 90 examines the electronic image to determine the original size 
of the pixel representing articles of the colloid specimen. Original size determination is 
utilized by program segment 98 to be further described hereinafter. Upon completion, 
program segment 90 passes control to program segment 92, by way of signal path 118. 
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Program segment 92 examines the electronic image and determines solid/liquid 
interface as manifesting a horizontal or vertical crystal growth. Upon completion, 
program segment 92 passes control to program segment 94, by way of signal path 120. 

Program segment 94 generates a series of frames of the electronic image. Upon 
completion, program segment 94 passes control to program segment 96, by way of signal 
path 122. 

Program segment 96 performs averaging of the frames to provide an image 
average over the series of frames so as to visually separate the solid and liquid portions of 
the electronic image. Upon completion, program segment 96 passes control to program 
segment 98, by way of signal path 124. 

Program segment 96 takes into account the fact that due to the solid region 
containing some of the same intensity values as the liquid regions, some erosion of the 
particle may occur. Program segment 98 compensates for the erosion by dilating the 
particles to approximate the original size. 

After the brightness slice operation performed by program segments 88, 90, 92 
and 94, the solid portion of the region is made up of many individual white pixels. A 
Find Particles algorithm, part of the algorithms 40, to be further described hereinafter, 
will only find particles that are 8-connected. A perfectly 8-connected object has a 
continuous perimeter that can be traced by traveling in 8 directions (up, down, left, right, 
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and diagonals) and at least one pixel separating the object from the image edge. For the 
method of Fig. 7, it is desired to dilate the white pixels to ensure that they will form a 
connected mass. In one practice of the present invention, the best dilation technique was 
a 7x7 low pass filter with 0 at the center and larger weights away from the center as 
shown in Table 1 . 
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The dilation produces a thin strip along the border of the image that is blacked out 
to ensure the solid region is separated from the image edge. 

With reference to Table 1, it is seen the dilation produces a square matrix of 
weights of size m (in this case 7). This weighting is applied to the neighborhood of the 
current pixel (the current pixel is centered) and is given as follows: 

w n Wu ... w lm 

W 2 1 W 22 
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Upon completion, program segment 98 passes control to program segment 100, 
shown in Fig. 7B, via signal path 126. 

Program segment 100 performs a threshold algorithm previously discussed with 
reference to brightness slicing. Upon completion, program segment 100 passes control to 
program segment 102, by way of signal path 128. 

Program segment 102 receives the dilated threshold treated electronic image of 
program segment 100. Program segment 102 then examines and displays the dilated 
threshold treated electronic image to visually identify solid and liquid portions, each 
manifesting a separate pattern and preferably also employs a fine particle technique that 
scans the image and identifies every non-background object. 

The find particle technique, serving as one of the algorithms 40, searches a 
grayscale image for non-background particles using an eight directional search (N, NE, 
E, SE, S, SW, W, NW) known in the art. The centroid of each particle is found using an 
intensity weighted center of mass algorithm also known in the art and part of the 
algorithms 40. 

The program segment 102 provides the examined dilated and threshold treated 
electronic image, and displays such on the display terminal 44 as represented by the 
image shown in Fig. 10. The image of Fig. 10 may be further enhanced by performance 
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of program segments 104 and 106. Upon completion, program segment 102 passes 
control to program segment 104, by way of signal path 129. 

Program segment 104 superimposes a contrast line 130 on the interface shape as 
shown in Fig. 1 1. To determine the interface 130, the largest particle found is determined 
to be the solid region. Depending on the previously specified orientation (horizontal or 
vertical growth) the topmost pixel in each column or the rightmost pixel in each row is 
taken as the solid/liquid interface. Upon completion, program segment 104 passes 
control to program segment 106, by way of signal path 132. 

Program segment 106 examines the contrast line to define data thereof. Upon 
completion, program segment 106 passes control, via signal path 134 to program segment 
108. 

Program segment 108 stores the data in computer 28, in particular the database for 
the examined colloidal hard shape sphere residing in the holder 14. 

Figs. 8, 9, 10, and 1 1 represent the images associated with the horizontal growth 
for the interface detection technique employing brightness slicing. This interface 
detection technique employing brightness slicing is also applicable to the vertical growth 
case and is shown in Figs. 12 and 13. 
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It should now be appreciated that the present invention provides two related 
techniques for the detection of the interface of the colloid hard sphere experiments. 
Because of the practice of the present invention, experiments that used to take weeks or 
even months to analyze manually can now be accomplished in a manner of minutes or 
hours for a longer experiment. 

Another method of the present invention related to automatically focusing on a 
specimen may be further described with reference to Fig. 14 composed of Figs. 14A and 
14B, each showing portions of the overall method 136. More particularly, Fig. 14A 
shows the overall method 136 for automatically focusing on a specimen as comprised of 
elements 138, 140, 142, 144, 146, and 148. Similarly, Fig. 14B shows the overall method 
136 for automatically focusing on a specimen as further comprised of elements 150, 152, 
154, and 156. 

As seen in Fig. 14 A, the routine 136 for automatically focusing on a specimen 
starts off with attaching a micro-positioner 22 A, shown in Fig. 1, to a Cartesian robotic 
system 22. The micro-positioner 22 A preferably has three stages: coarse; medium; and 
fine. The micro-positioner 22A has a vertical level having a top limit which causes the 
camera of the video microscope 12 to be situated by predetermined distance, which is the 
furthest distance possible from the target that is, the specimen that is held in holder 14. 
This predetermined distance depends on the type of robotic system being used. It can be 
as small as a millimeter and as long as a few feet. The practice of the invention used a 
system with a maximum distance of 1 foot and a system with a maximum distance of 2 
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inches and both worked well. The micro-positioner 22A, preferably has ten separate 
steps for each stage and the step size for the coarse stage is about 10mm. Upon 
completion of segment 138, the method of Fig. 14 passes control to segment 138, via 
signal path 158. 

Segment 140 signifies that the specimen to be examined is placed in a holder and 
then segment 140 passes control to program segment 142, by way of signal path 160. 

Program segment 142 causes the video microscope 12 to generate an electronic 
image comprises spatial frequencies, wherein any spatial frequency represents the rate of 
brightness transition. 

Images generated by the video microscope 12 are made up of spatial frequencies. 
A spatial frequency is the rate of brightness transition (the rate at which spatial details 
cycle from light to dark and back to light). The Fast Fourier Transform is one method 
used to decompose an image into its fundamental frequency components. There is a 
magnitude and a phase value associated with each frequency component. For each 
image, a focus value is computed by taking the Fast Fourier Transform (FFT) of the 
largest possible portion of the center of the image. After the FFT operation, the image 
moves from the spatial domain to the frequency (or Fourier) domain. When an image is 
displayed in the frequency domain, the magnitudes of the frequency components that 
make up the image are also displayed. 



24 



For each pixel in the original image, there is one value displayed in the frequency 
transform image. The display of the magnitudes in the frequency domain is arranged 
symmetrically around the center of the image, with horizontal frequencies on the 
horizontal axis, vertical frequencies on the vertical axis, and diagonal frequencies off the 
axes. The zero-frequency point of the display is in the center. Frequencies increase 
outward from the center of the image to the maximum rate at the outermost point. The 
magnitude of each frequency component is indicated by the darkness of the pixel at each 
location in the image: the darker the pixel, the greater the magnitude of the corresponding 
spatial frequency. 

Upon completion, program segment 142 passes control to program segment 144, 
by way of signal path 162. 

Program segment 144 performs a Fast Fourier Transform on the electronic image 
so as to decompose the electronic image into its fundamental components with each 
component having magnitude and phase qualities. The FFT performance transforms the 
electronic image from a spatial domain to a frequency domain. Upon completion, 
program segment 144 passes control to program segment 146, by way of signal path 164. 

Program segment 146 examines the FFT electronic images by displaying the 
magnitude of the frequency components. A blurry, poorly focused image (see Fig. 15) is 
made up of low frequencies positioned near the center of the FFT. This results in a 
compact FFT pattern, with dark pixels near the center and a lot of white area (see Fig. 
16). A correctly focused image of the same scene (see Fig. 17) will contain higher 
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frequencies, resulting in a dispersed FFT pattern, with dark pixels spread over a wider 
area and less white background (see Fig. 18). Upon completion, program segment 146 
passes control to program segment 148, by way of signal path 166. 

Program segment 148 positions the vertical position of a micro-positioner 22 A to 
its top level so that the camera is located away from the specimen by its first 
predetermined distance which is also the maximum distance in which the camera can be 
positioned away from the specimen. Upon completion, program segment 148 passes 
control, via signal path 168, to program segment 150 shown on Fig. 14B. 

Program segment 150 positions by separate steps, preferably 10, the micro- 
positioner 22 A so that the camera moves downward and toward the specimen and a 
predetermined number of step sizes, preferably 10mm, associated with the coarse stage of 
the micro-positioner, while at the same time examining that the FFT pattern being 
displayed becomes further dispersed, in a manner as shown in Fig. 18, as the camera 
moves downward by each of the coarse steps. Upon completion, program segment 150 
passes control to program segment 152, by way of signal path 170. 

Program segment 152 determines the sum of a magnitude of the FFT pattern 
while at the same time determining that the sum of the magnitude of the FFT pattern 
decreases as the camera moves downward at each coarse step. The magnitude over the 
FFT are summed for each image as the image becomes more focussed. The position 
where the focus value is at lowest is considered to be most in-focus. It is preferred that 
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the step size of 10mm be decreased by a factor of 10 and that program segments 150 and 
1 52 be repeated. During such adjustments, the micro-positioner 22A moves such that the 
current-in-focus positioner is centered in a new range. Upon completion, program 
segment 152 passes control to program segment 154, by way of signal path 172. 

5 

Program segment 154 repeats program segments 150 and 152, but sets the micro- 
positioner 22A to its medium stage before such repeating. Upon completion, program 
segment 154 passes control to program segment 156, by way of signal path 174. 

10 Program segment 156 repeats program segments 150 and 152, but now sets the 

micro-positioner 22A to its fine stage before such repeating is performed. 

Upon completion of program segment 156, the method 136 produces an image 
that is ideally focussed to its maximum contrast. 

15 

The positioning of the micro-positioner 22A may be accomplished by a neural 
network routine appropriately programmed so that it sets its associated elements in a 
random manner and, based upon operational feedback these elements are molded into the 
pattern required to provide the required results, that is, an ideally focused image with 
20 maximum contrast. 

Another embodiment of the present invention that comprises a routine for cell 
identification and classification includes the matrix/statistics of one or more cells, may be 
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further described with reference to Fig. 19 composed of Figs. 19A and 19B, wherein Fig. 
19A shows the overall method 176 as comprised of the elements 178, 180, 182, 184, 186, 
188 and, similarly, Fig. 19B shows the overall method 176 as further comprised of 
elements 190, 194, 196, 198, 200, 202, and 204. 

Cell identification for biomedical applications is done all around the world on a 
daily basis by thousands of humans looking through microscopes for hours at a time. The 
related technique of the present invention utilizes human information and incorporates 
that into an intelligent machine vision technique in order to develop a system capable of 
using only the best techniques for cell identification. The present invention provides a 
technique for the automatic detection of cell candidates for patch clamping, known in the 
art. This technique automatically identifies all possible cells in an image (Good & Bad). 
This technique also provides metrics/statistics, known in the art, on cells such as 
intensity-weighted center of mass (centroid), major axis length, minor axis length, area, 
roundness, smoothness, elongation, shape, partial cells, and cell orientation. Fig. 20 
illustrates an example of a potential patch clamping candidate. The method 176 for cell 
identification may be further described again with reference to Fig. 19 A. 

As seen in Fig. 19 A, the overall method 176 starts with segment 178 wherein the 
one or more cells serving as a specimen are placed in the holder. Upon completion, 
segment 178 sequences to program segment 180, by way of signal path 206. 
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Program segment 180 generates a series of electronic images of the specimen and 
then passes control to program segment 182, by way of signal path 208. 

Program segment 182 generates a histogram having bars, which are proportional 
5 to the values of the intensity of the series of electronic images. In order to identify and 
classify cells, it is necessary to pre-process the images to allow isolation of the cells from 
their background. For the patch clamping images related to program segment 176, two 
parallel pre-processing paths are provided. The first path related to program segment 182 
is a histogram illustration shown in Fig. 20. Upon completion, program segment 182 
10 passes control to program segment 184, by way of signal path 210. 

Program segment 184 performs a Sobel filtering operation of the histogram so as 
to highlight the edges of the changes in the density of the histogram. The Sobel filter 
highlights the edges by finding changes in intensity thereof. The Sobel filtering 
15 technique is known in the art. Fig. 21 shows the image after receiving the histogram's 
treatment and the Sobel filtering operation. Upon completion, program segment 184 
passes control to program segment 186, by way of signal path 212. 

Program segment 186 performs a threshold algorithm previously described with 
20 reference to the brightness slicing technique of the present invention. 

The threshold algorithm, serving as one of the algorithms 40, eliminates small 
intensity changes in the image provided to program segment 186. Program segment 186 
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clearly identifies particle boundaries, but does not always preserve cell connectivity (cell 
parameters often have gaps). This is corrected by allowing program segment 186 to pass 
control, via signal path 214, to program segment 188. 

Program segment 188 causes the video microscope 12 to generate an electronic 
image, which is now used for the second path of the overall process 176 of Fig. 19. Upon 
completion, program segment 188 passes control, via signal path 216, to program 
segment 190 shown on Fig. 19B. 

Program segment 190 performs a variance filtering operation, serving as one of the 
algorithms 40, to produce an image thereof shown in Fig. 22. As the filter passes over 

I N 

the image, the variance, s, is computed as s 2 = — Y (x, - x) 2 , where N is the number of 

pixels and x is the average in each neighborhood. Upon completion, program segment 
190 passes, via signal path 218, control to program segment 194. 

Program segment 194 examines the variance of the neighborhood of the pixels. If 
the neighborhood of the center pixel has a low variance, the area is basically constant and 
the center of the pixel is set to black. More particularly, and with reference to Fig. 23, if 
the center pixel having a lip-like shape has a low variance, then the center of the lip-like 
shape is set to black. If the neighborhood has a high variance, intensity values are 
changing over the neighborhood and the center pixel is left unchanged. This approach 
preserves the connectivity of the cells and has no corruption of cell date, but is also 
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includes some of the background as part of the cell. Upon completion, program segment 
194 passes control to program segment 196, by way of signal path 220. 

Program segment 196 combines the treated histogram program segment 186 with 
the colored or uncolored center pixel of program segment 194 that results in an image 
shown in Fig. 24. Upon completion, 196 passes control to program segment 198, by way 
of signal path 222. 

Program segment 198 performs a find pixel technique to identify non-background 
objects of the combined histogram of program segment 196 in a manner as previously 
described with reference to Fig. 7. 

Program segment 198, as well as other program segments of the present 
invention, may employ a clean technique that cleans the image using a center-deleted 
spike filter modified to remove noise not associated with particles. The clean technique 
is arranged to work on images with a black background. The mask size designates the 
extent of the neighborhood around the current pixel that will determine its output value. 
A clutter threshold, known in the art, represents the cutoff for the sum of pixel values 
lying within the mask region (except the center pixel). If the sum is less than the clutter 
threshold, the center pixel will be set to 0. A significance threshold represents the cutoff 
for a given pixel value to be considered significant. If the center pixel is below the 
significance threshold, it will be set to 0. When cleaning a binary image, the significance 
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threshold is not necessary since all pixels are either white (255) or black (0), so it is set to 

0. 



Upon completion, program segment 198 passes control to program segment 200, 
by way of signal path 224. 



Program segment 200 performs an overlay technique, known in the art, on the 
combined histogram to block out background objects and provides an image such as that 
shown in Fig. 24. Upon completion, program segment 200 passes control to program 
segment 202, by way of signal path 226. 



Program segment 202 analyzes the non-background objects to produce matrix of 
the specimen being held in the holder. 

TABLE 2 



#id 


X 


1 


major 


minor 


area 


elong 


round 


smooth theta thin 


prtl 


1 


517.1 


18.3 


14.9 


8.8 


117 


0.589 


2.169 


0.184 


98 F 


F 


2 


289.3 


274.9 


124.5 


66.5 


4882 


0.534 


0.178 


0.143 


22.6 F 


F 


3 


84.2 


310 


3.5 


2.7 


16 


0.776 


0.88 


0.154 


31.7F 


F 


4 


388.4 


313.4 


6.4 


3.2 


24 


0.495 


0.93 


0.183 


37 F 


F 


5 


227.2 


330.7 


16.7 


11.2 


155 


0.67 


0.967 


0.16 


114.9F 


F 


6 


602.9 


369.1 


4.7 


3.2 


21 


0.674 


1.817 


0.115 


135 F 


F 


7 


594.5 


373.9 


11.9 


6.1 


62 


0.515 


0.697 


0.19 


137.6 F 


F 


8 


91.1 


400.9 


37.8 


9.4 


202 


0.249 


0.232 


0.161 


42.2 F 


F 


9 


151.9 


451.3 


16.4 


7.6 


99 


0.462 


0.535 


0.169 


26. IF 


F 


10 


158.7 


463.4 


13.7 


9.9 


106 


0.723 


0.898 


0.174 


43.5 F 


F 



Based on the information listed in Table 2, program segment 202 can classify an 
ideal patch clamping candidate by its major axis, minor axis, area, elongation, roundness, 
smoothness, theta (orientation with respect to a coordinate system), thinness, and whether 
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the object is completely (F) contained in the field of view or only partially contained 
(partial). Program segment 202 passes control, via signal path 228, to program segment 
202. 



Program segment 204 compares the produced metrics of Table 2 against known 
metrics to identify and classify the one or more cells being held in the holder 14. 



The result from our technique practicing the method 176 of Fig. 2 and starting 
with the image of Fig. 25 yielded the following Table 3. 

TABLE 3 



#id 


X 


I 


maior 


minor 


area 


elong 


round 


smooth theta thin 


pit! 


1 


72.1 


89.4 


191.8 


126.7 


12491 


0.661 


0.077 


0.143 


137 F 


T 


2 


597.3 


33.1 


49.9 


44.9 


844 


0.899 


0.18 


0.162 


1.2 F 


T 


3 


564.9 


11.8 


6.8 


3.7 


27 


0.537 


0.908 


0.203 


40.3 F 


F 


4 


575.4 


28.2 


17.3 


9.3 


114 


0.536 


0.449 


0.1 


83.3 F 


F 


5 


559.1 


21.3 


4 


2.5 


15 


0.615 


0.943 


0.159 


80.8 F 


F 


6 


629.1 


41 


3.4 


2.3 


13 


0.676 


0.99 


0.205 


103.3 F 


T 


7 


628.6 


51.2 


12 


4 


38 


0.333 


0.489 


0.144 


90 T 


T 


8 


594.8 


97.5 


16 


14 


175 


0.876 


0.884 


0.168 


4.7 F 


F 


9 


168.7 


107.6 


4 


4 


19 


1 


1.775 


0.167 


OF 


F 


10 


93.1 


238.7 


247.1 


117.3 


14243 


0.475 


0.093 


0.139 


44.7 T 


F 


11 


598.9 


128.9 


35.8 


14.6 


277 


0.409 


0.323 


0.185 


99.4 F 


F 


12 


614.3 


132.3 


22.1 


5.5 


121 


0.25 


0.361 


0.161 


76.5 F 


F 


13 


623.2 


146.4 


5.4 


1.9 


15 


0.346 


0.619 


0.135 


74 F 


F 


14 


629.6 


158.2 


3.2 


1 


7 


0.302 


0.752 


0.188 


74.5 F 


T 


15 


342.4 


223.3 


121.6 


88.4 


6417 


0.727 


0.301 


0.141 


160.6 F 


F 


16 


623.8 


185.2 


6.7 


4.1 


26 


0.618 


0.965 


0.2 


58.3 F 


F 


17 


600.6 


371 


5.1 


3.8 


21 


0.744 


1.805 


0.188 


125.8 F 


F 


18 


593.4 


376.9 


6.7 


5 


31 


0.739 


1.269 


0.203 


137.4 F 


F 


19 


100.8 


392.3 


8 


4.6 


34 


0.572 


0.994 


0.147 


36 F 


F 


20 


149.5 


453 


9.4 


5 


50 


0.535 


0.845 


0.125 


26.6 F 


F 


21 


155.4 


464.9 


5.4 


2.2 


15 


0.396 


0.75 


0.25 


5.7 F 


F 



By looking at the results listed in Table 3, it is clear that cell 15 of Table 3 is the 
ideal candidate because it most closely matches the metrics identified for an ideal patch 
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clamping candidate cell. This is a significant advancement in the field of patch clamping 
because we now have the ability to automatically scan through a sample specimen and 
identify ideal cell candidates with no human interaction. The technique has proven to be 
more efficient over time than an actual human who has to look at each and every cell 
sample over a period of hours. 

A second example resulting from the patch clamping technique of the present invention is 
illustrated in Fig. 26. 

Another method of the practice of the present invention provides for a routine that 
identifies one or more cells in an area of interest, which cell is closest to the center of the 
area interest, the nearest neighbors of each cell within the area of interest and tracks 
movements of the cells within the area of interest routine for cell identification. The 
method may be further described with reference to Fig. 27 for an overall program 230 
having elements 232, 234, 236, and 238. 

As seen in Fig. 27, the overall method starts with segment 232 which signifies one 
or more cells, serving as a specimen, are placed in the holder 14. Upon completion, 
segment 232 gives control to program segment 234, by way of signal path 240. 

Program segment 234 generates an electronic image, such as that shown in Fig. 
28, of the one or more cells, each having a dark outer ring. Upon completion, program 
segment 234 passes control to program segment 236, by way of signal path 242. 
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Program segment 236 examines the dark outer ring of each cell to determine the 
shape and size of the one or more cells. By concentrating on the dark outer ring of each 
cell, it allows for determination of the shape and size of the cell, but prior to the 
examination the cells may have an image such as that of Fig. 29. Upon completion, 
program segment 236 passes control to program segment 238, by way of signal path 244. 

Program segment 238 performs an intensity weighted center of mass technique to 
determine the position of each cell within the area of interest. The intensity weighted 
center of mass technique is known in the art and serves as one of the algorithms 40. 

The intensity weighted center of mass technique determines the position of each 
cell to each other. The program segment 238 then labels the cell based on the 
information of each center with a labeled process image such as that shown in Fig. 30 and 
the original image with the cells labeled shown in Fig. 3 1 . The center of each cell is 
identified in Fig. 30 by reference number 246 A. The partial particles shown in Fig. 3 1 by 
reference number 246B can be classified as such and then perimeters within the image 
can be determined. 

It should now be appreciated that the practice of the present invention provides a 
procedure that seeks to maintain high accuracy and repeatable techniques for cell 
identification in the medical industry. It is contemplated that this technique also reduces, 
by a significant amount, the amount of lost time human error typically creates. 
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Another embodiment provided by the machine vision techniques of the present 
invention comprises a routine for evaluating surfaces of devices, such as semiconductor 
devices and is shown in Fig. 32, and an overall program 248 comprised of elements 250, 
252, 254, and 256. 

The technique illustrated by Fig. 32 utilizes an ideal or representative surface and 
treats this ideal surface as the standard for evaluating remaining samples. Each surface to 
be analyzed is compared to the standard and if it meets certain tolerance metrics, it will 
be approved for use. The goal in this technique is to categorize good or acceptable 
surfaces vs. bad or unacceptable surfaces. By classifying the surface defects, the present 
invention labels and categorizes each defect as it appears in an area of interest. The 
surface classification technique works by isolating any defects on a given surface and 
transforming the pixel information into an objective measure of surface quality that can 
be compared to the ideal case. 

As seen in Fig. 32, the program 248 is initiated by a program segment 250, which 
provides an electronic image of a desired surface of a device. The electronic image of 
program segment 252 possesses an intensity representative of the critical value of the 
desired surface. Upon completion, program segment 250 passes control to program 
segment 252, by way of signal path 258. 
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Program segment 252 represents the placement of the device in the holder so that 
its surface faces the camera of the video microscope 12. Upon completion, program 
segment 252 passes control to program segment 254, by way of signal path 260. 

Program segment 254 generates an electronic image of the surface of the device 
in the holder. This electronic image includes a morphological gradient operation (known 
in the art), which is performed on the image to find feature edges. This operation creates 
an edge-enhanced image of the boundaries of the features in the image, such as that 
shown on the lower portion of Fig. 34. The outlines represent the portions of the image 
where the steep dark-to-bright and dark-to-dark transition occur. Then a center deleted 
spike filter, previously discussed with reference to Fig. 19, is applied to remove one-pixel 
duration variations and small features of the pixel. The pixels in the resulting image are 
summed to provide a measure of surface quality. Surface quality for each case is then 
compared to the critical value represented by the ideal surface. Fig. 33 represents a 
smooth or ideal surface, whereas Figs. 34 and 35 show examples of the same type of 
surface exhibiting defects. The surface quality value for each case is compared to the 
critical value represented by the ideal case of Fig. 33. If a surface has an intensity value 
greater than the ideal surface as in the cases of Figs. 34 and 35, it may be classified as 
being defective. The critical value can be adjusted for various types of surfaces. 

A second ideal surface is shown in Fig. 36, which represents an ideal case, and 
Figs. 36 and 37 show defect samples of the same type of surface related to Fig. 36. To 
accommodate all of this analysis, program segment 254 passes control to program 



37 



segment 256, by way of signal path 262. Program segment 256 completes all the 
comparisons of the intensity of a desired surface and the surface being evaluated. 

The surface classification technique represented by Fig. 27 allows an ideal surface 
5 to be stored in an image database and retrieved for comparison of similar surfaces that 
can be used for in-line process inspection or for an automatic design system to accept or 
reject a surface based on the ideal surface. 

It should now be appreciated that the practice of the present invention provides a 
CMIS that can conduct experiments using microscopes without the need of constant 
10 monitoring. Further, the CMIS allows for the analysis of microscopic experiments 

automatically and remotely. Still further, the CMIS provides a utilization of intelligence 
allowed by adapted neural networks to cause the desired feed-back resulting in accurate 
and repeatable results. 

15 It should be still further appreciated that the CMIS scans, finds areas of interest, 

focuses and acquires images automatically. It should be further appreciated that the 
CMIS allows for the evaluation of surfaces of devices, such as semiconductors to be 
accurately and quickly determined. 

20 The invention has been described with reference to preferred embodiments and 

alternates thereof It is believed that many modifications and alteration to the 
embodiments as described herein will readily suggest themselves to those skilled in the 
art upon reading and understanding the detailed description of the invention. It is 
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intended to include all modifications and alterations insofar as they come within the 
scope of the present invention. 
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